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SYSTEMS AND METHODS FOR PROVIDING ONLINE 
FAST SPEAKER ADAPTATION IN SPEECH RECOGNITION 

RELATED APPLICATION 
[0001] . This application claims priority under 35 U.S.C § 1 19 based on U.S. Provisional 
Application No. 60/419,214, filed October 17, 2002, the disclosure of which is incorporated 
herein by reference. 

[0002] This application is related to the concurrently-filed U.S. application (Docket No. 02- 

4018), serial number , titled "Systems and Methods for Classifying Audio into Broad 

Phoneme Classes," which is incorporated herein by reference. 

BACKGROUND OF THE INVENTION 

Field of the Invention 

[0003] The present invention relates generally to speech recognition and, more particularly, 
to systems and methods for providing online fast speaker adaptation in a speech recognition 
system. 

Description of Related Art 

[0004] Speaker information can be used in a speaker-independent speech recognition system 
to adapt the speech recognition process and, thereby, improve the accuracy of the speech 
recognition results. The goal of speaker adaptation is not to permanently change the models used 
by the speech recognition system, but to temporarily adapt the model so that it better represents 
the words of the speaker whose speech the speech recognition system is trying to recognize. 
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[0005] Fig. 1 is a functional block diagram of a conventional speech recognition system. The 
system includes a first decoder 1 10, speaker clustering logic 120, a transformation estimator 130, 
and a second decoder 140. Decoder 1 10 receives segments of speech (e.g., utterances) from an 
audio source. Decoder 1 10 decodes the speech segments using speaker-independent acoustic 
models (e.g., Hidden Markov Models (HMMs)) to produce transcriptions of the speech 
segments. At this point there is no speaker adaptation. 

[0006] Speaker clustering logic 120 groups speech segments together by speaker. In other 
words, speaker clustering logic 120 determines which speech segments were produced from the 
same speaker and groups these segments into clusters. Transformation estimator 130 estimates a 
transformation matrix for each cluster that, when applied to the audio, moves the audio data 
closer to the model. To do this, transformation estimator 130 typically collects data, estimates 
the transformation matrix from the data, and applies the transformation matrix to the models. 
[0007] Transformation estimator 130 estimates the transformation matrix by accumulating 
statistics using forward and backward alignment, which is a probabilistic technique for 
estimating a transformation based on marginal contributions of certain probabilities. Typically, 
transformation estimator 130 uses straight cepstrals, a first order difference (i.e., a first derivative 
that indicates the differences between adjacent cepstrals), and a second order difference (i.e., a 
second derivative that indicates how fast the cepstrals are changing). Transformation estimator 
130 uses fourteen features from each group to build a 42 x 42 transformation matrix that is 
applied to the audio data. 

[0008] Decoder 140 then decodes the speech segments with adaptation using the 
transformation matrices. Because decoder 140 adapts based on the speaker, decoder 140 
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produces transcriptions of the speech segments that are better than the transcriptions produced by 
decoder 110. 

[0009] The conventional system has several drawbacks. For example, the conventional 
system requires two passes of the decoder (decoder 110 and decoder 140). In other words, each 
speech segment is decoded twice. Further, the speaker clustering performed by speaker 
clustering logic 120 is a non-causal function because all of the speech segments must be 
available before the clustering can be accomplished. These factors result in delays that make a 
real-time operation impractical. 

[0010] As a result, there exists a need for systems and methods that improve the performance 
of speaker adaptation in a speech recognition system. 



SUMMARY OF THE INVENTION 
[0011] Systems and methods consistent with the present invention provide adaptation based 
on speaker turns in a streamlined and efficient manner that may be used in a real-time 
environment. Within each speaker turn, the speech may be segmented into small chunks. 
During decoding of the first chunk, no adaptation may be performed. Adaptation may then be 
performed on the second and subsequent chunks within the same speaker turn using a 
transformation matrix that is estimated from the transcriptions resulting from previous chunks. 
[0012] In one aspect consistent with the principles of the invention, a system performs 
speaker adaptation when performing speech recognition. The system receives an audio segment 
and identifies the audio segment as a first audio segment or a subsequent audio segment 
associated with a speaker turn. The system then decodes the audio segment to generate a 
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transcription associated with the first audio segment when the audio segment is the first audio 
segment and estimates a transformation matrix based on the transcription associated with the first 
audio segment. The system decodes the audio segment using the transformation matrix to 
generate a transcription associated with the subsequent audio segment when the audio segment is 
the subsequent audio segment. 

[0013] In another aspect consistent with the principles of the invention, a decoder within a 
speech recognition system is provided. The decoder includes a forward decoding stage, a 
backward decoding stage, and a rescoring stage. The forward decoding stage, the backward 
decoding stage, and/or the rescoring stage is/are configured to receive an audio segment, identify 
the audio segment as a first audio segment or a subsequent audio segment associated with a 
speaker turn, decode the audio segment to generate a transcription associated with the first audio 
segment when the audio segment is the first audio segment, estimate a transformation matrix 
based on the transcription associated with the first audio segment, and decode the audio segment 
using the transformation matrix to generate a transcription associated with the subsequent audio 
segment when the audio segment is the subsequent audio segment. 
[0014] In yet another aspect, consistent with the principles of the invention, a speech 
recognition system is provided. The speech recognition system includes speaker change 
detection logic and a decoder. The speaker change detection logic receives audio segments and 
identifies boundaries between speakers associated with the audio segments as speaker turns. The 
decoder receives, from the speaker change detection logic, one of the audio segments as a 
received audio segment associated with one of the speaker turns, and identifies the received 
audio segment as a first audio segment or a subsequent audio segment associated with the 
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speaker turn. The decoder decodes the received audio segment to generate a transcription 
associated with the first audio segment when the received audio segment is the first audio 
segment and constructs a transformation matrix based on the transcription associated with the 
first audio segment. The decoder decodes the received audio segment using the transformation 
matrix to generate a transcription associated with the subsequent audio segment when the 
received audio segment is the subsequent audio segment. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0015] The accompanying drawings, which are incorporated in and constitute a part of this 
specification, illustrate the invention and, together with the description, explain the invention. In 
the drawings, 

[0016] Fig. 1 is a functional block diagram of a conventional speech recognition system; 
[0017] Fig. 2 is a functional block diagram of an exemplary speech recognition system in 
which systems and methods consistent with the present invention may be implemented; 
[0018] Fig. 3 is an exemplary functional block diagram of the decoder of Fig. 2 according to 
an implementation consistent with the principles of the invention; 

[0019] Fig. 4 is a flowchart of exemplary processing by one or more of the elements of Fig. 3 
according to an implementation consistent with the present invention; and 
[0020] Fig. 5 is an exemplary diagram of a transformation matrix according to an 
implementation consistent with the present invention. 
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DETAILED DESCRIPTION 
[0021] The following detailed description of the invention refers to the accompanying 
drawings. The same reference numbers in different drawings may identify the same or similar 
elements. Also, the following detailed description does not limit the invention. Instead, the 
scope of the invention is defined by the appended claims and equivalents. 
[0022] Systems and methods consistent with the present invention provide online speaker 
adaptation based on speaker turns. Within each speaker turn, the speech may be divided into 
small segments. During decoding, speaker adaptation may be performed on second and 
subsequent segments within the same speaker turn using a transformation matrix that is estimated 
from the transcriptions generated from previous segments. 

EXEMPLARY SYSTEM 
[0023] Fig. 2 is a diagram of an exemplary speech recognition system 200 in which systems 
and methods consistent with the present invention may be implemented. System 200 may 
include phone class decoder 210, speaker change detection logic 220, and decoder 230. Phone 
class decoder 210 may include logic that receives an input audio stream and performs initial 
signal processing functions on the audio stream. For example, phone class decoder 210 may 
convert the audio stream into a frequency domain signal and generate cepstral features for the 
audio data. 

[0024] Phone class decoder 210 may also classify the audio stream into a number of broad 
classes, such as vowels, fricatives, narrowband, wideband, coughing, gender, and silence. Phone 
class decoder 210 may generate an output signal that indicates the classification for a particular 
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portion of the audio stream. The output signal may be a continuous signal or a signal that 

classifies the audio stream in predetermined length segments (e.g., 30 second chunks). 

[0025] One implementation of phone class decoder 210 is described in copending U.S. Patent 

Application, Serial No. 10/ (Docket No. 02-4018), entitled "Systems and Methods 

for Classifying Audio into Broad Phoneme Classes," filed on . Another 

implementation of phone class decoder 210 is described in D. Liu et al., "Fast Speaker Change 
Detection for Broadcast News Transcription and Indexing," Proceedings of Eurospeech 99, 
September 1999, pp. 1031-1034. The preceding two documents are incorporated herein by 
reference. 

[0026] Speaker change detection logic 220 may locate the boundaries between speakers (i.e., 
speaker turns) in the input audio stream. Speaker change detection logic 220 may also divide the 
audio stream into approximately four second segments within the speaker turns. To identify the 
speaker turns, speaker change detection logic 220 may implement techniques similar to those 
described in J. Makhoul et al., "Speech and Language Technologies for Audio Indexing and 
Retrieval," Proceedings of the IEEE, Vol. 88, No. 8, August 2000, pp. 1338-1353, which is 
incorporated herein by reference. Speaker change detection logic 220 may output audio data as 
approximately four second segments with indications of changes in speaker turns. 
[0027] Decoder 230 may perform speaker adaptation to generate transcription results based 
on the audio segments and the speaker turns. Fig. 3 is an exemplary functional block diagram of 
decoder 230 according to an implementation consistent with the principles of the invention. 
Decoder 230 may include forward decoding stage 310, backward decoding stage 320, and 
rescoring stage 330. Forward decoding stage 310, backward decoding stage 320, and rescoring 
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stage 330 may use acoustic models, such as Hidden Markov Models (HMMs), to generate 
transcripts of the audio segments. 

[0028] Forward decoding stage 310 may include a very robust HMM and perform a very fast 
rough match. Forward decoding stage 310 may attempt to narrow the search space for a correct 
answer, but does not try to find the correct answer. Instead, forward decoding stage 310 simply 
makes sure not to lose the correct answer. Backward decoding stage 320 may include a more 
complex HMM and perform a slower, more accurate match. Backward decoding stage 320 starts 
with a smaller search space (i.e., the search space narrowed by forward decoding stage 310) and 
narrows the search space even further. 

[0029] Rescoring stage 330 may include an even more complex HMM and perform a slower 
accurate match. Rescoring stage 330 starts with a much smaller search space (i.e., the search 
space narrowed by forward decoding stage 310 and backward decoding stage 320) and selects the 
best answer (i.e., the correct word). Each of these stages progressively narrows the search space 
to find the correct word. 

[0030] Speaker adaptation may apply to any or all of these stages. Fig. 4 is a flowchart of 
exemplary processing for speaker adaptation according to an implementation consistent with the 
principles of the invention. The following acts will be described as occurring in forward 
decoding stage 310. It should be understood, however, that these acts are also performed in 
backward decoding stage 320 and/or rescoring stage 330. 

[0031] Processing may begin when forward decoding stage 310 receives an audio segment 
(e.g., an utterance) from speaker change detection logic 220. Forward decoding stage 310 may 
determine whether this audio segment is a first audio segment of a speaker turn (act 410). First 
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audio segments may be identified by speaker change indications provided by speaker change 
detection logic 220. If the audio segment is a first audio segment, then forward decoding stage 
310 may decode the audio segment using one or more speaker-independent HMMs to obtain 
transcription results (act 420). 

[0032] Forward decoding stage 3 1 0 may then reset the transformation matrix for the current 
speaker to a predetermined state (act 430). The transformation matrix may be reset because 
information regarding a different speaker is typically not useful when trying to decode speech 
from the current speaker. 

[0033] Forward decoding stage 310 may then estimate the transformation matrix based on 
transcription results and, possibly, other features associated with the first audio segment (act 
440). Unlike conventional approaches that use fourteen features associated with each of the 
straight cepstrals, the first order difference, and the second order difference to build a 42 x 42 
matrix, first decoding stage 310 may use only the fourteen features associated with the straight 
cepstrals to build a 14 x 14 matrix according to an implementation consistent with the principles 
of the invention. In this way, forward decoding stage 310 need only estimate a 14 x 14 matrix 
instead of a 42 x 42 matrix. To construct the 42 x 42 matrix, forward decoding stage 310 may 
replicate the 14 x 14 matrix three times. 

[0034] Fig. 5 is an exemplary diagram of a transformation matrix according to an 
implementation consistent with the present invention. As shown in Fig. 5, the 14 x 14 matrix is 
replicated along the diagonal of the 42 x 42 matrix. The rest of the matrix may be filled with a 
predetermined value, such as zero. In other implementations consistent with the present 
invention, the 14 x 14 matrix appears in other locations within the 42 x 42 matrix. 
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[0035] Returning to Fig. 4, when identifying the values for the transformation matrix, 
forward decoding stage 310 may use a known statistical alignment technique, such as the Viterbi 
alignment. Unlike conventional approaches that use probabilistic techniques, the Viterbi 
alignment is a statistical alignment that aligns the answers with the audio data, scores them, and 
takes the highest score. Forward decoding stage 310 may then apply the transformation matrix to 
the models in a conventional manner (act 450). 

[0036] When the audio segment received by forward decoding stage 3 10 is not the first audio 
segment in a speaker turn, then forward decoding stage 310 may decode the audio segment using 
the transformation matrix to obtain transcription results (act 460). Forward decoding stage 310 
may then use the transcription results to reestimate the transformation matrix (act 440). The 
transformation matrix is a cumulative matrix that is incrementally improved with every audio 
segment in a speaker turn. The matrix may be reestimated with each new segment from the same 
speaker. This serves to incrementally improve the transformation matrix and, thus, the 
transcription results. 

[0037] Returning to Fig. 3, forward decoding stage 310, backward decoding stage 320, and 
rescoring stage 330 perform incremental speaker adaptation according to an implementation 
consistent with the principles of the invention. Forward decoding stage 310 receives audio 
segments from speaker change detection logic 220 in approximately four second utterances. 
When an audio segment corresponds to a first audio segment in a speaker turn, forward decoding 
stage 310 may reset its transformation matrix. Forward decoding stage 310 may then process the 
audio segments in a manner similar to that described above and output them to backward 
decoding stage 320 in approximately four second utterances. 
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[0038] Backward decoding stage 320 may receive the audio segments from forward decoding 
stage 310. When an audio segment corresponds to a first audio segment in a speaker turn, 
backward decoding stage 320 may reset its transformation matrix. Backward decoding stage 320 
may then process the audio segments in a manner similar to that described above and output 
them to rescoring stage 330 in approximately four second utterances. Rescoring stage 330 may 
receive the audio segments from backward decoding stage 320. When an audio segment 
corresponds to a first audio segment in a speaker turn, rescoring stage 330 may reset its 
transformation matrix. Rescoring stage 330 may then process the audio segments in a manner 
similar to that described above and output transcription results. 

[0039] In an alternate implementation consistent with the principles of the invention, 
backward decoding stage 320 and/or rescoring stage 330 may perform speaker-turn-based 
adaptation instead of the incremental adaptation described above. According to speaker-turn- 
based adaptation, backward decoding stage 320 and/or rescoring stage 330 may wait for the 
entire speaker turn to complete in the previous stage before decoding the audio segments of the 
speaker turn. In this way, the transcription results generated by the previous stage(s) may be used 
to construct the transformation matrix. The transformation matrix may then be used to decode 
the audio segments, including the first audio segment, in the speaker turn. 
[0040] In yet another implementation consistent with the principles of the invention, 
backward decoding stage 320 and/or rescoring stage 330 may use information from the previous 
stage(s) when performing adaptation. For example, when backward decoding stage 320 
processes a second audio segment in a speaker turn, backward decoding stage 320 may use 
information (e.g., transcription results) generated by forward decoding stage 310 when 
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processing the first and second audio segments and information (e.g., transcription results) 
generated by backward decoding stage 320 when processing the first audio segment to construct 
the transformation matrix. 

CONCLUSION 

[0041] Systems and methods consistent with the present invention provide online speaker 
adaptation based on speaker turns. Within each speaker turn, the speech may be segmented into 
small chunks. Speaker adaptation may be performed on second and subsequent chunks within 
the same speaker turn using a transformation matrix that is estimated from the transcriptions 
resulting from previous chunks. 

[0042] The following benefits may be obtained by an online speaker adaptation process 
according to an implementation consistent with the principles of the invention. For example, the 
speaker adaptation process involves only a single pass of the decoder (see Fig. 4), unlike 
conventional approaches that require two passes of the decoder. The speaker adaptation process 
is a real-time streamlined process, unlike conventional approaches that use speaker clustering and 
its inherent delays. The speaker adaptation process uses only straight cepstrals to estimate a 14 x 
14 matrix instead of conventional approaches that required estimation of a 42 x 42 matrix. The 
speaker adaptation process may use a statistical alignment process, such as Viterbi alignment, to 
perform the transformation. Viterbi alignment is faster and involves less computations than 
conventional approaches that use probabilistic techniques. 

[0043] The foregoing description of preferred embodiments of the present invention provides 
illustration and description, but is not intended to be exhaustive or to limit the invention to the 
precise form disclosed. Modifications and variations are possible in light of the above teachings 
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or may be acquired from practice of the invention. For example, while a series of acts has been 
described with regard to Fig. 4, the order of the acts may differ in other implementations 
consistent with the principles of the invention. Also, non-dependent acts may be performed in 
parallel. 

[0044] Further, certain portions of the invention have been described as "logic" that performs 
one or more functions. This logic may include hardware, such as an application specific 
integrated circuit or a field programmable gate array, software, or a combination of hardware and 
software. 

[0045] No element, act, or instruction used in the description of the present application 
should be construed as critical or essential to the invention unless explicitly described as such. 
Also, as used herein, the article "a" is intended to include one or more items. Where only one 
item is intended, the term "one" or similar language is used. The scope of the invention is 
defined by the claims and their equivalents. 
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